Neuron-by-Neuron Quantization for Efficient Low-Bit QNN Training

Quantized neural networks (QNNs) are widely used to achieve computationally efficient solutions recognition problems. Overall, eight-bit QNNs have almost the same accuracy as full-precision networks, but working several times faster. However, with lower quantization levels demonstrate inferior in comparison their classical analogs. To solve this issue, a number of quantization-aware training (QAT) approaches were proposed. In paper, we study QAT for two- linear schemes and propose new combined approach: neuron-by-neuron straight-through estimator (STE) gradient forwarding. It is suitable quantizations widths eliminates significant drops during training, which results better final QNN. We experimentally evaluate our approach on CIFAR-10 ImageNet classification show that it comparable other four eight bits outperforms some them two three while being easier implement. For example, proposed three-bit dataset 73.2% accuracy, baseline direct layer-by-layer result 71.4% 67.2% respectively. The two-bit ResNet18 63.69% 61.55% baseline.